…lanner (apache#1904)

* feat(scheduler): broadcast small build side of SortMergeJoinExec

Add ballista.optimizer.broadcast_sort_merge_join_enabled (default false). When
enabled, the static distributed planner converts a SortMergeJoinExec whose
smaller side is under broadcast_join_threshold_bytes into a CollectLeft hash
join and lowers the build side as a broadcast stage, reusing the existing
hash-join broadcast path. Redundant input sorts are dropped during conversion.

TPC-H SF10 (AQE off): ~16% faster across the suite (q8 1.6x, q11 1.5x, q2 1.4x),
identical row counts.

* feat(config): default broadcast_sort_merge_join_enabled to true

Enable the SortMergeJoinExec broadcast conversion by default so small build
sides are broadcast without opt-in (~16% on TPC-H SF10, AQE off). Disable the
flag explicitly in the sort-merge-join client test so it still exercises the
plain SortMergeJoinExec execution path.

* test: assert SMJ broadcast plans via string snapshots

Rewrite the three SortMergeJoinExec broadcast tests to assert the
stage plan as a string via the assert_plan! macro instead of walking
the plan tree with manual downcasts, making the expected plan explicit
and the tests easier to read.

…dopt clobbered 53->54 features

The 54.0.0 merge suffered a delete/modify pathology: files the fork had
deleted relative to base 53.0.0 silently stayed deleted whenever upstream
had not touched them during the 53->54 cycle, and several conflict
resolutions kept stale fork-side file versions. This commit repairs it.

Tree repairs (restored from upstream 54.0.0):
- ballista-cli: tui/domain/{mod,executors}.rs, ui/main/jobs/dot_parser.rs,
  exec.rs — the crate did not compile with default features
- examples/standalone-broadcast-join.rs (Cargo.toml still declared it)
- scheduler/src/api/ (routes.rs + handlers.rs + mod.rs) and display.rs
- docs python/ dir, example notebooks, test_jupyter.py, TUI screenshots,
  benchmarks/tpch-gen.sh, vendored datafusion*.proto stubs

Re-adopted upstream features the merge clobbered:
- apache#1911 partition pruning (under disable-stage-plan-cache) + tests
- apache#1968 shuffle-read metrics, mapped onto the Spice fetch pipeline, and
  child-operator metrics in writer plan displays
- apache#1995 session-keyed runtime cache wired into the executor (plus the
  upstream --memory-pool-size FairSpillPool option)
- apache#1982 selective intermediate-stage shuffle cleanup end-to-end (also
  applied to the in-memory shuffle manager)
- apache#1949 failed-task REST surfacing + apache#1818 CORS/--disable-rest-api,
  TaskManager::get_all_jobs / get_job_config, JobState::get_all_jobs,
  ExecutorManager::get_executors_state (upstream names)
- apache#1900/apache#1904 static-planner broadcast promotion (maybe_promote_to_
  broadcast, CollectLeft demotion guard, SMJ->hash conversion, broadcast
  stage lowering, broadcast_join_threshold_bytes config) + 12 tests
- apache#1999 task duration in finished-task log
- execute_physical_plan client entry point (apache#1924/apache#1941)
- ballista.client.io_retries_times / io_retry_wait_time_ms, wired into
  the evict-and-retry fetch loop (fork-preserving defaults: 1 retry, 0ms)

Cleanups and adaptations:
- deleted dead executor/src/client_pool.rs, orphaned AQE optimizer files,
  committed .pending-snap artifacts
- shuffle.remote_read_prefer_flight now defaults to true: the block-IO
  transport cannot serve sort-based shuffle (default-on), so the old
  default pair was incoherent; removed the block-IO sort-shuffle test
  cases accordingly
- regenerated TPC-H plan-stability goldens (TopK single-stage merge from
  the fork's #28.1 was not reflected in the goldens taken from upstream)
- SPICE_FORK_CHANGES.md updated to match reality (#61/#62a rows, apache#1951
  disposition, repair-commit inventory)